An image may represent a view of a scene as captured from the viewpoint of a camera. In some cases there may be more than one camera capturing different views of a scene. However, there will be some viewpoints of the scene which do not correspond to any of the camera viewpoints. The image may be a frame of a video sequence. Techniques such as Free-viewpoint video rendering (FVVR) allow a novel view of a scene to be generated based on a set of multiple views of the scene from multiple camera viewpoints. The cameras are preferably calibrated and synchronized with each other so that the views of the scene can be combined correctly.
Based on the different views of the scene, a model of the scene geometry may be constructed, for example using Multiple-View Stereo (MVS), and a texture may be formed which can be applied to the model. The texture can be formed by projectively texturing the scene geometry with the original images and blending the projected images. The model, with the texture, can then be used to render the scene from a rendering viewpoint which may, or may not, be the same as one of the camera viewpoints. As well as recreating a “real-world” scene from a rendering viewpoint, the content of the real-world scene may be mixed with computer-generated content.
There are a number of issues which may need to be considered when generating a novel viewpoint of a scene. For example, relighting of the scene can be difficult. Textures extracted from images (e.g. frames of a video sequence) captured by cameras have implicit real-world lighting information, such that lighting artifacts are present (i.e. “baked-in”) in the textures.
One way of addressing the problem of how to relight the textures for a novel viewpoint is to control the lighting of the scene at the time when the cameras capture the different views of the scene. For example, diffuse lighting can be used in the initial video capture to avoid creating excess shaded areas and specularities that will damage the plausibility of the scenes rendered using extracted textures. The effects of lighting changes may be automatically addressed, but this may require an active lighting arrangement, in which the scene is captured under a variety of calibrated lighting conditions, in order to deduce the material properties of the textures. However, relighting scenes with arbitrary lighting arrangements is considerably more challenging. Similar challenges apply to relighting textures with arbitrary lighting arrangements irrespective of how the textures were formed from captured images of scenes, for example when only one camera is used to capture an image of a scene from a single camera viewpoint and/or when the rendering viewpoint is the same as one of the camera viewpoints.